Skip to content

feat(diagnostics): surface twd-js failure diagnostics in the run summary - #20

Merged
kevinccbsg merged 1 commit into
mainfrom
feat/failure-diagnostics
Sep 5, 2026
Merged

feat(diagnostics): surface twd-js failure diagnostics in the run summary#20
kevinccbsg merged 1 commit into
mainfrom
feat/failure-diagnostics

Conversation

@kevinccbsg

@kevinccbsg kevinccbsg commented Sep 5, 2026

Copy link
Copy Markdown
Member

A failing test now reports which mock rules never fired, above the error message:

  Failed tests (1):
    × Inquiries > should list the company group inquiries
      mock rules  6/7 triggered — catalog never requested
      AssertionError: expected 0 rows (at http://localhost:5173/cg-1/settings/catalog)

Above the error, not below, for upstream's reason: a twd-js failure message can carry a full accessible-roles dump that would bury the block.

Consumes the snapshot added by BRIKEV/twd#335.

Needs no change to twd-js

NEXT-DIAGNOSTICS.md planned to render the block inside the page, which required twd-js to expose its formatter as window.__twdFormatDiagnostics — new public surface on the library's engine entry, and a blocking maintainer decision.

That turned out to be unnecessary. The snapshot is plain data, so the in-page onFail just carries test.diagnostics out with the result entry and rendering happens in Node.

It's the better split regardless:

  • This package already owns its terminal style (the relay-style run block) and shares no rendering with twd-js. The sidebar removal in twd's c46b17e confirms the two surfaces are diverging on purpose.
  • The stable contract between the packages is the data shape, not the rendering. A shared formatter would have coupled them on the part meant to differ.
  • The snapshot lands in the run report as structured data, so shard artifacts and anything reading run.json get it too — not just the terminal.

The location row is deliberately dropped

This resolves the open question in NEXT-DIAGNOSTICS.md. Every failure message here already ends in (at <href>), and the full href is strictly more informative than the snapshot's pathname + search + hash. Printing both would say the same thing twice, so the CLI keeps the href and renders only what it did not already have.

No schema bump

diagnostics is additive and optional on tests[]. runReport.js spreads it through and mergeReports.js carries it across the merge without either knowing about it. Bumping would make an additive field needlessly loud, given the version guard is already exact-match.

Safe to merge before the twd-js release

Output is byte-identical to today when no snapshot is present, which is exactly what shipped twd-js 1.9.0 sends. The block simply appears once diagnostics land on npm.

Verification

Unit:

  • Formatter — every branch: no snapshot, no rules, all triggered, one miss, several misses, the five-item cap, a missing untriggered array.
  • Summary — rows land above the error and indent to the error column; byte-identical output when no snapshot is present; nothing printed for a test that passed on retry.
  • In-page hand-off — the real onFail is driven against a stub runner. page.evaluate is mocked everywhere else, so this callback would otherwise never execute in the suite.
  • Shard round-trip — snapshot survives the merge and a JSON round-trip, embedded newlines in error intact.

Live, against a real browser:

  • Shipped twd-js 1.9.0 (no snapshot): a real failing test prints exactly as before, no block, no crash, exit code 1.
  • twd-js main after #335, built and installed from a local pack: the populated path renders for real — single-alias row, multi-alias list, and no row at all for a test that registered no rules.

456 tests pass (was 439). Coverage 96.13% → 97.08%.

Upstream caveat found while verifying — twd-js, not this package

collectDiagnostics() reads getRequestMockRules(), the global rule registry, with no per-test scoping. Nothing in twd-js resets it between tests; twd.clearRequestMockRules() is public but calling it is up to the app.

So in an app that does not clear rules in an afterEach, aliases accumulate across the run and each failing test is blamed for every rule registered before it. Three failing tests in one test-example-app run:

× ... one mock rule never requested        mock rules  0/1 triggered — neverRequestedCatalog never requested
× ... several mock rules never requested   mock rules  0/4 triggered — 4 never requested
× ... no mock rules registered at all      mock rules  0/4 triggered — 4 never requested

The third registers zero mocks. Run in isolation it correctly prints no row, which confirms the rendering is right and the input is not.

Worth fixing in twd-js before release — it inverts the feature, pointing at four aliases unrelated to the failure. Nothing to change here: twd-cli renders faithfully whatever the snapshot contains.

🤖 Generated with Claude Code

@kevinccbsg
feat(diagnostics): surface twd-js failure diagnostics in the run summary
af05ccb 
A failing test now reports which mock rules never fired, above the error
message:

      mock rules  6/7 triggered — catalog never requested
      AssertionError: expected 0 rows (at http://localhost:5173/cg-1/...)

Above the error, not below, because a twd-js failure message can carry a
full accessible-roles dump that would bury it.

The planned approach needed twd-js to expose its formatter on `window` so
the in-page `onFail` could reach it — new public surface on the library,
and a blocking decision. That is not necessary: the snapshot is plain
data, so `onFail` carries `test.diagnostics` out with the result entry and
rendering happens in Node.

Better split regardless. This package already owns its terminal style and
shares no rendering with twd-js — a divergence the upstream sidebar
removal confirms is deliberate. The stable contract between the two is the
data shape, not the rendering. It also puts the snapshot in the run report
as structured data, so shard artifacts and anything reading run.json get
it too, not just the terminal.

Only the mock-rule signal is rendered. The snapshot's `location` row is
dropped: every failure message here already ends in `(at <href>)`, and the
full href is strictly more informative than pathname + search + hash.
Printing both would say the same thing twice.

No schema bump. `diagnostics` is additive and optional on tests[];
runReport spreads it through and mergeReports carries it across the merge
without either knowing about it.

Verified beyond the mocked suite: the in-page onFail is driven against a
stub runner, since page.evaluate is mocked everywhere else and would never
otherwise execute it. Output is byte-identical to before when no snapshot
is present, and a real failing test against the shipped twd-js 1.9.0
prints exactly as it always has, exit code 1.

twd-js has not released diagnostics yet (PR #335 open), so the populated
path is unit-tested only and degrades to today's output until it ships.

456 tests pass, coverage 96.13% -> 97.08%.
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

TWD Contract Validation

Spec Passed Failed Warnings Mode
./contracts/users-3.0.json 2 3 1 warn
./contracts/posts-3.1.json 2 2 0 warn
./contracts/products-3.0.json 13 23 2 warn
./contracts/events-3.1.json 6 13 0 warn

23 passed · 41 failed · 3 warnings · 1 skipped

Failed validations

./contracts/users-3.0.json

  • GET /users/{userId} (200) — mock getUserNoAddress — in "Contract Validation - Mismatches > should fail: missing nested address field"
    • response.address: missing required property "address"
  • GET /users/{userId} (200) — mock getUserBadAddress — in "Contract Validation - Mismatches > should fail: nested address missing required city"
    • response.address.city: missing required property "city"
    • response.address.country: missing required property "country"
  • GET /users/{userId} (200) — mock getUserBadRole — in "Contract Validation - Mismatches > should fail: oneOf role with invalid variant"
    • response.role: oneOf best match (branch 2 of 2) failed: must be one of: "viewer"

./contracts/posts-3.1.json

  • GET /posts/{postId} (200) — mock getPostNoAuthor — in "Contract Validation - Mismatches > should fail: post missing nested author object"
    • response.author: missing required property "author"
  • GET /posts/{postId} (200) — mock getPostBadMeta — in "Contract Validation - Mismatches > should fail: post oneOf metadata matches neither variant"
    • response.metadata: oneOf best match (branch 1 of 2) failed: missing required property "category", unexpected property "duration", must be one of: "article"

./contracts/products-3.0.json

  • GET /products (200) — mock getProductEmptyName — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: empty name violates minLength"
    • response[0].name: must NOT have fewer than 1 characters
  • GET /products (200) — mock getProductBadSku — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid SKU pattern"
    • response[0].sku: must match pattern "^[A-Z]{2,4}-\d{4,8}$"
  • GET /products (200) — mock getProductBadUuid — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uuid format for id"
    • response[0].id: must match format "uuid"
  • GET /products (200) — mock getProductBadDateTime — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date-time format"
    • response[0].createdAt: must match format "date-time"
  • GET /products (200) — mock getProductBadDate — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date format"
    • response[0].releaseDate: must match format "date"
  • GET /products (200) — mock getProductBadEmail — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid email format"
    • response[0].contactEmail: must match format "email"
  • GET /products (200) — mock getProductBadUri — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uri format"
    • response[0].website: must match format "uri"
  • GET /products (200) — mock getProductBadIp — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv4 format"
    • response[0].serverIp: must match format "ipv4"
  • GET /products (200) — mock getProductBadIpV6 — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv6 format"
    • response[0].serverIpV6: must match format "ipv6"
  • GET /products (200) — mock getProductZeroPrice — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: price of 0 violates exclusiveMinimum"
    • response[0].price: must be > 0
  • GET /products (200) — mock getProductNegQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: negative quantity violates minimum"
    • response[0].quantity: must be >= 0
  • GET /products (200) — mock getProductOverQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: quantity exceeds maximum"
    • response[0].quantity: must be <= 999999
  • GET /products (200) — mock getProductBadWeight — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: weight not multipleOf 0.01"
    • response[0].weight: must be multiple of 0.01
  • GET /products (200) — mock getProductBadRating — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: rating above maximum (5)"
    • response[0].rating: must be <= 5
  • GET /products (200) — mock getProductBadCurrency — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for currency"
    • response[0].currency: must be one of: "USD", "EUR", "GBP", "JPY"
  • GET /products (200) — mock getProductBadCategory — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for category"
    • response[0].category: must be one of: "electronics", "clothing", "food", "books", "toys"
  • GET /products (200) — mock getProductBadBool — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: string value for boolean inStock"
    • response[0].inStock: expected boolean, got string
  • GET /products (200) — mock getProductDupTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: duplicate tags violates uniqueItems"
    • response[0].tags: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /products (200) — mock getProductTooManyTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: tags exceeds maxItems (10)"
    • response[0].tags: must NOT have more than 10 items
  • GET /products (200) — mock getProductBadMeta — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: non-string value in metadata additionalProperties"
    • response[0].metadata.count: expected string, got number
  • GET /settings (200) — mock getSettingsBadExtra — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: extra property on Settings (additionalProperties: false)"
    • response.extraField: unexpected property "extraField"
  • GET /settings (200) — mock getSettingsBadLang — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid language pattern in Settings"
    • response.language: must match pattern "^[a-z]{2}(-[A-Z]{2})?$"
  • GET /products (200) — mock getProductBadNullable — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response[0].description: expected string,null, got number

./contracts/events-3.1.json

  • GET /events (200) — mock getEventsEmpty — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty events array violates minItems (1)"
    • response: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventShortName — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: event name too short (minLength: 3)"
    • response[0].name: must NOT have fewer than 3 characters
  • GET /events (200) — mock getEventBadDate — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid date-time format for startDate"
    • response[0].startDate: must match format "date-time"
  • GET /events (200) — mock getEventFloatId — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: float value for integer id"
    • response[0].id: expected integer, got number
    • response[0].id: must match format "int64"
  • GET /events (200) — mock getEventBadBool — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: number value for boolean active"
    • response[0].active: expected boolean, got number
  • GET /events (200) — mock getEventBadStatus — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid enum value for status"
    • response[0].status: must be one of: "draft", "published", "archived"
  • GET /events (200) — mock getEventScoreMax — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: score at exclusiveMaximum boundary (100)"
    • response[0].score: must be < 100
  • GET /events (200) — mock getEventLowPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority below minimum (1)"
    • response[0].priority: must be >= 1
  • GET /events (200) — mock getEventHighPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority above maximum (5)"
    • response[0].priority: must be <= 5
  • GET /events (200) — mock getEventDupAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: duplicate attendees violates uniqueItems"
    • response[0].attendees: must NOT have duplicate items (items ## 1 and 0 are identical)
  • GET /events (200) — mock getEventNoAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty attendees array violates minItems (1)"
    • response[0].attendees: must NOT have fewer than 1 items
  • GET /events (200) — mock getEventBadAttendee — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid email format in attendees"
    • response[0].attendees[0]: must match format "email"
  • GET /events/{eventId} (200) — mock getEventBadNullable — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
    • response.description: expected string,null, got number

View full report →

@kevinccbsg
kevinccbsg merged commit cd5792a into main Sep 5, 2026
6 checks passed
This was referenced Sep 6, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kevinccbsg